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Foreword 


In  2008,  the  Office  of  Naval  Research  (ONR)  funded  the  Personnel 
Integration  of  Selection,  Classification,  Evaluations,  and  Surveys  (PISCES)  effort 
to  address  three  deficiencies  seen  in  the  personnel  arena:  (1)  reliance  on 
independent,  single  use  measures  for  assessing  the  Navy  Total  Force;  (2)  lack  of 
measures  to  classify  and  select  individuals  for  team  assignments;  and  (3)  lack  of 
integration  of  tools  to  measure  and  predict  Navy  Total  Force  performance.  One 
objective  of  PISCES  is  to  alleviate  the  first  deficiency  by  developing  a  set  of  data 
collections  alternatives  that  could  be  used,  then  matching  the  best  collection 
method  with  the  type  of  information  needed. 

Lightning  Poll  is  one  of  the  data  collection  alternatives  developed.  It  is  a  text 
messaging-based  option  for  data  collection,  adapted  from  civilian  use  of  text 
messaging  for  participation  in  television  shows,  among  other  things.  This  report 
is  the  first  of  three  to  outline  the  Lightning  Poll  portion  of  the  Data  Collection 
Alternatives  project,  and  discusses  the  reasons  behind  considering  this  as  an 
alternative  as  well  as  the  mechanics  of  creating  a  Lightning  Poll.  The  second 
report  will  provide  results  of  the  first  Lightning  Poll,  conducted  with  Navy 
Recruiters  who  are  authorized  to  use  text  messaging  in  their  recruiting  duties  and 
therefore  may  have  experience  with  text  messaging.  The  third  report  contains 
results  of  the  second  Lightning  Poll  as  well  as  the  accompanying  Telework  and 
Technology  Survey,  and  suggests  other  possible  data  collection  alternatives  based 
upon  the  results  of  the  survey. 

The  authors  thank  ONR  for  their  sponsorship.  They  also  gratefully 
acknowledge  members  of  the  two  information  technology  departments  who  were 
instrumental  in  gaining  approval  to  purchase  and  use  the  Lightning  Poll  software. 
Within  the  Navy  Personnel  Command  Chief  Information  Officer  Department,  Mr. 
Dell  Turner  and  LCDR  Joseph  Namie;  and  within  NPRST’s  Research  Information 
Systems  Management  Office,  ITCS  Billy  Freeman  and  ETC  Derek  Brown  were 
particularly  helpful  at  the  outset  of  this  project. 


David  L.  Alderton,  Ph.D. 

Director 
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I  ntroduction 


The  possibility  of  near  instantaneous,  scientifically-reliable  survey  data 
gathered  from  a  geographically  dispersed  group  at  low  cost  is  intriguing  to  many 
survey  researchers.  Historically,  this  has  not  been  possible;  however,  recent 
technological  advances  have  made  it  feasible.  Web-based  surveys  are  close  to  the 
ideal,  but  typically  require  access  to  a  computer,  which  necessitates  additional 
time  for  those  who  are  away  from  their  home  or  office.  The  increased  use  of  cell 
phones  with  internet  access  alleviates  this  issue,  but  still  requires  a  potential 
respondent  to  take  action  such  as  checking  e-mail  (i.e.,  requires  the  person  to 
“pull”  information  from  an  e-mail  server)  to  learn  that  they  have  been  selected  to 
participate  in  a  survey.  Surveys  through  text  messaging  eliminate  this  step  by 
“pushing”  the  survey  directly  to  the  respondent  through  their  cell  phone  and 
providing  questions  almost  instantly. 

Realizing  the  possibilities  of  surveys  through  text  messaging,  the  Navy 
conducted  the  first  known  Navy  text  message-based  survey  in  summer  2008. 

This  report  explains  the  history  of  text  messaging  and  some  of  the  current  uses  in 
the  civilian  population,  before  describing  the  four  stages  that  NPRST  used  to 
develop  their  Lightning  Poll,  or  survey-by-text-message,  capability.  The  first 
stage,  Planning,  includes  recommendations  based  upon  lessons  learned  that  are 
useful  regardless  of  the  software  package  and/or  contractor  used  to  send  text 
message  surveys.  The  second  two  stages,  Programming  and  Administration, 
provide  steps  and  screen  shots  specific  to  the  Lightning  Poll  software.  The  final 
stage,  Analysis,  includes  some  software  specific  information  as  well  as 
suggestions  for  analyzing  the  data  returned  and  sample  SPSS  syntax.  A 
discussion  of  future  directions  of  text  messaging  surveys  is  also  included. 

Description  of  Text  Messaging 

Because  a  cell  phone  does  not  restrict  its  user’s  mobility,  the  location  of  a  cell 
phone  changes  constantly.  However,  a  great  deal  of  information  must  be  used  to 
maintain  connectivity  regardless  of  the  location.  Within  the  Global  System  for 
Mobile  (GSM)  family  of  protocols,  a  portion  of  the  information  used  to  maintain 
that  connectivity  is  dedicated  to  Short  Message  Service  (SMS),  otherwise  known 
as  text  messaging.  The  first  SMS  was  sent  in  1992  (Grenville,  2007),  and  usage 
has  been  climbing  since,  with  June  2008  statistics  showing  that  75  billion  text 
messages  are  sent  every  month  (CTIA,  2009).  Usage  by  younger  populations 
(which  can  be  defined  as  those  born  since  1980  (Pew  Research  Center  for  the 
People  and  the  Press,  2007))  is  highest,  and  some  have  made  news  for  the  large 
number  of  messages  that  they  send  within  one  month  (Mah,  2009).  Results  from 
Pew  indicate  that  60  percent  of  those  between  18  and  29  years  old  who  have  a  cell 
phone  or  personal  digital  assistant  send  or  receive  text  messages  in  a  typical  day, 
decreasing  to  32  percent  of  30-49-year-olds,  14  percent  of  50-64-year-olds,  and 
2  percent  of  those  65  and  over  (Horrigan,  2008). 
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In  this  regard,  the  Navy  population  does  not  appear  much  different  from  the 
civilian  population  at  large.  Findings  show  that  in  2005,  61  percent  of  enlisted 
personnel  sent  a  text  message  each  week,  with  7  percent  sending  50+  messages 
per  week  (Uriell  &  Schultz,  2007).  More  recent  results  show  that  64  percent  of 
enlisted  personnel  and  59  percent  of  officer  personnel  in  Generation  Y  (also 
known  as  Generation  Next,  which  includes  those  born  from  1966-1980  (Pew 
Research  Center  for  the  People  and  the  Press,  2007))  sent  or  received  text 
messages  in  the  last  day,  decreasing  to  31  percent  and  32  percent  of  those  in  the 
Boomer  generation  (defined  as  those  born  in  1965  and  before  (Pew  Research 
Center  for  the  People  and  the  Press,  2007)),  enlisted  and  officers  respectively 
(Schultz,  Newell,  Whittam,  &  Uriell,  2008). 

Text  messages  are  generally  created  on  a  cell  phone  using  the  QWERTY 
keypad  (if  available)  or  the  numeric  keypad,  as  shown  in  Figure  1.  Using  the 
QWERTY  keypad  is  like  using  a  computer  keyboard,  with  each  button  assigned  to 
a  particular  letter.  For  phones  that  do  not  have  the  QWERTY  keypad,  messages 
are  typed  by  pressing  the  numeric  keypad  either  1,  2,  or  3  times  (or  4  times  for 
numbers  7  and  9)  to  arrive  at  the  appropriate  letter.  For  example,  an  “X”  would 
require  pressing  the  9  key  twice.  For  more  recent  phones,  there  is  also  a  function 
that  will  attempt  to  automatically  complete  the  word  being  typed  based  upon  the 
numeric  keys  pressed,  allowing  the  users  to  only  have  to  type  each  numeric  key 
once,  however  sometimes  the  automatic  completion  needs  to  be  checked  since 
“GO”  and  “IN”  use  the  same  keys. 


Figure  1.  Telephone  keypad. 


Because  text  messages  are  sent  as  part  of  the  processing  overhead  for  cell 
phone  connectivity  (Stress,  2008;  see  Dryburgh  &  Hewitt,  2004,  for  a  detailed 
description),  they  generally  arrive  shortly  after  they  are  sent,  although  this  is 
dependent  upon  the  cell  phone  reception.  The  cost  is  based  upon  the  cell  phone 
plan  of  the  user;  some  plans  allow  for  unlimited  use  for  a  monthly  fee  while 
others  may  charge  10  or  20  cents  per  message  (incoming  and  outgoing). 
Messages  are  at  most  160  characters,  although  some  cell  phone  providers  utilize 
some  of  this  space;  therefore,  140  characters  is  the  suggested  maximum  per 
message. 
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The  length  of  the  message  has  not  deterred  their  use  for  a  variety  of  reasons 
beyond  the  initial  use  of  contacting  friends  and  family  (McCarthy,  2009).  Text 
messages  are  now  used  to  coordinate  events,  spread  information  about  political 
candidates  (Graff,  2008;  Kennedy,  2009),  and  gather  donations  (Kim,  2009). 
They  are  also  used  in  emergencies  for  sending  out  city-wide  emergency 
notifications  (Chang-Yen,  2009),  school  closure  messages,  Amber  Alerts  (Hodgin, 
2009);  to  interact  with  businesses,  such  as  the  voting  in  American  Idol; 
advertising  (Richtel,  2009);  receiving  coupons  (Bowman,  2007);  ordering 
products  and  food  (The  Daily  Green,  2008);  and  even  programming  home 
electronics  such  as  TiVo  (Falcone,  2009). 


Planning 


Planning  ahead  is  a  key  step  in  the  process  of  doing  a  successful  text 
messaging  survey.  In  particular,  a  number  of  issues  need  to  be  considered  early  in 
the  process,  including  how  to  gather  cell  phone  numbers  of  potential  respondents, 
how  to  notify  potential  respondents  of  their  legal  rights,  and  how  to  design 
question-and-answer  combinations  that  fit  within  the  140  character  limit  of  text 
messages. 

Legality 

When  planning  to  do  a  text  message  survey,  there  are  two  laws  to  be  aware  of: 
the  Telephone  Consumer  Protection  Act  (TCPA)  of  1991  (available  online  from 
the  Library  of  Congress  at  http :  /  /thomas  .loc.  gov/cgi- 

bin/query/z?ci02:S.i462.ENR:')  and  the  Controlling  the  Assault  of  Non-Soli  cited 
Pornography  and  Marketing  (CAN-SPAM)  Act  of  2003  (available  from  the 
Government  Printing  Office  at  http://frwebgate.access.gpo.gov/cgi- 
bin/getdoc.cgi?dbname=io8  cong  public  laws&docid=f:publi87.io8),  the 

second  of  which  enabled  the  national  Do  Not  Call  list.  The  TCPA  indicates  that 
the  sending  of  text  messages  to  cell  phones  using  an  automatic  telephone  dialing 
system  is  prohibited  without  the  express  consent  of  the  person  receiving  the 
message.  The  CAN-SPAM  Act  does  not  include  messaging  to  those  with  a  pre¬ 
existing  relationship.  While  it  could  be  argued  that  a  member  of  the  Navy  has  a 
pre-existing  relationship  with  the  Navy,  it  is  recommended  that  you  receive  the 
consent  of  potential  respondents  prior  to  conducting  a  survey  using  text 
messaging  since  voluntary  participation  is  a  requirement  for  all  Navy  surveys  (see 
OPNAVINST  5300.8  series). 

Additionally,  many  are  concerned  about  phishing  (emails  from  seemingly 
legitimate  sources  such  as  banks  that  contain  links  to  fake  websites  that  request 
sensitive  information  such  as  account  number  or  password)  and  identity  theft,  so 
they  may  be  concerned  about  giving  any  information  to  an  unknown  source,  be  it 
through  a  phone  call,  an  e-mail,  a  website,  or  a  text  message.  In  fact,  the  2009 
Navy  training  about  Information  Assurance  states  that  Navy  personnel  should 
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not  participate  in  text  messaging  surveys,  so  prior  notification  to  respondents  of 
the  legitimacy  of  the  survey  is  key  to  having  a  successful  data  collection.  See  the 
sampling  section  below  for  more  details  on  ways  to  gather  cell  phone  numbers 
and,  thereby  obtain,  consent. 

Software/  Hardware 

There  are  many  software  products  and  services  available  that  can  provide  the 
tools  to  conduct  text  message  surveys.  Many  of  those  available  at  the  time  of  the 
initial  Navy  text  message  survey  required  that  the  company  have  access  to  the  cell 
phone  numbers  (e.g.,  cell  phone  numbers  were  uploaded  to  a  company  web 
server,  as  well  as  question/answer  messages),  which  would  be  a  concern 
considering  Navy  and  DoD  regulations  and  concerns  about  protecting  personally- 
identifiable  information  (PII)  (e.g.,  name,  SSN,  phone  number).  Therefore, 
Interactive  Messaging  Unlimited  (IMU)  Pro  was  purchased  from  Pangolin  Laser 
Systems,  Inc.  This  software  provides  the  flexibility  to  conduct  a  text  messaging 
survey  yourself  and  keep  all  data  in-house.  As  part  of  the  IMU  Pro  package,  a 
wireless  modem  (at  the  time  of  purchase,  the  modem  provided  was  the 
MultiModem  GPRS,  model  MTCBA-G-U-F2,  from  MutiTech  Systems)  is  included 
to  enable  the  in-house  computer  to  wirelessly  send  text  messages  to  cell  phones. 

Because  of  Navy  security  concerns  about  wireless  computing,  a  standalone 
Dell  Precision  T3400  was  purchased  with  a  2.66GHz  processor  and  4GB  of 
SDRAM.  This  Windows  XP  system  was  placed  into  a  secured  environment  to 
protect  the  data. 

A  Subscriber  Identity  Module  (SIM)  card  was  purchased  through  the  General 
Services  Administration  (GSA).  SIM  cards  are  now  used  in  cell  phones  as 
additional  memory  and  to  store  the  cell  phone’s  number,  increasing  the 
portability  of  the  owner’s  information  from  one  phone  to  another.  Since  the 
project  only  required  a  SIM  card  that  was  able  to  send  text  messages  and  not 
voice  transmissions,  the  contract  included  unlimited  text  messaging  and  no  voice 
capabilities.  The  SIM  card  was  inserted  into  the  modem  and  used  to  transmit 
incoming  and  outgoing  messages. 

IMU  Pro  is  programmed  to  automatically  hang  up  on  incoming  voice  calls, 
which  can  lead  to  frustration  for  a  potential  respondent  if  they  attempt  to  call  the 
phone  number  from  which  they  received  a  text  message.  Call  forwarding,  caller- 
ID,  and/or  working  with  the  cellular  carrier  are  options  that  would  allow  those 
receiving  the  messages  to  know  who  sent  them. 

Sample 

Currently,  an  accessable  database  of  personally-owned  cell  phone  numbers  of 
the  Navy  Total  Force  does  not  exist  (and  databases  that  may  collect  this 
information  would  be  for  emergency  contact  use  only).  There  are  databases  for 
Navy-provided  cell  phone  numbers,  but  this  may  not  include  the  population  of 
interest.  At  present,  there  is  no  standard  or  preferred  way  to  obtain  personal  cell 
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phone  numbers.  Some  suggestions  include  creating  a  registration  page  on  a 
website  where  potential  respondents  can  provide  their  demographic  information 
and  cell  phone  number,  sending  a  Naval  message  to  commands  of  interest  asking 
respondents  to  send  a  text  message  to  the  survey  phone  number  in  order  to  start 
the  survey,  or  asking  respondents  to  provide  their  cell  number  as  part  of  a  larger 
survey. 

Since  gathering  phone  numbers  might  be  the  hardest  part  of  the  process,  a 
series  of  surveys  could  be  more  effective,  such  as  those  that  use  a  panel  format. 
This  would  allow  for  a  single  notification,  but  multiple  collections  of  data.  Doing 
a  long-term  panel  might  require  consideration  of  methods  to  refresh  the  panel.  A 
text  message  panel  has  not  yet  been  established  within  the  Navy,  so  these  issues 
are  not  addressed  here. 

Some  respondents  may  be  aboard  a  deployed  vessel  (i.e.,  Navy  ship,  or 
submarine)  or  overseas  and  may  experience  difficulties  with  text  messaging.  At 
present,  it  is  unclear  if  text  messaging  is  possible  aboard  a  deployed  Navy  ship 
(i.e.,  distance  limitations  from  cell  phone  transmitters  may  inhibit  text 
messaging),  but  those  assigned  to  a  ship/submarine  may  be  pier-side  and  thus 
able  to  participate  in  a  text  message  survey. 

Surveying  overseas  respondents  presents  another  challenge.  Overseas  cell 
phone  providers  may  not  have  an  agreement  with  a  U.S.  provider  to  enable 
international  text  messaging.  Table  l  contains  a  listing  of  participating 
international  providers  for  AT&T  as  well  as  for  Verizon,  as  obtained  from  their 
respective  websites  on  15  January  2009;  only  those  overseas  respondents  who 
used  one  of  the  AT&T  partners  could  have  participated,  since  AT&T  is  the  SIM 
card  provider  for  the  Lightning  Poll.  OCONUS  phone  numbers  were  not  used  in 
either  of  the  Lightning  Polls  conducted  by  NPRST  to  date. 
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Table  1 

I  nternational  Participating  Providers  for  AT&T  and  Verizon 


AT&T 

Verizon 

Bahrain 

Batelco 

Guam 

Hafatel 

Guamcell 

Pulse  Mobile 

Hafatel 

ITE  Guam 

1  celand 

Landssiminn 

Iceland  Telecom 

Nova 

Viking  Telecom 

Og  Vodafone 
Viking  Telecom 


Italy 

H3G 

Vodafone 

WIND 

Vodafone 

Japan 

DoCoMo 

Vodafone 

DoCoMo 

KDDI 

Tuka 

Vodafone 

Spain 

France  Telecom  Espana  SA 
Telefonica  Moviles 

Vodafone 

Xfera  Moviles 

Vodafone 

United  Kingdom 

Hutchison  3G 

Orange 

02 

Vodafone 

Orange 

02 

T-Mobile 

Vodafone 

Note:  I  nformation  obtained  from  company  websites  on  15  J  anuary  2009. 


Regardless  of  the  method  for  obtaining  cell  phone  numbers,  researchers  must 
have  proper  Navy  survey  (Report  Control  Symbol  (RCS))  and,  if  applicable, 
Institutional  Review  Board  (IRB)  approvals  prior  to  conducting  any  Navy  survey. 
For  more  information  on  RCS  (examples,  frequently  asked  questions,  etc.)  and 
IRB  requirements,  visit  the  Navy  Survey  Policy  website  at 
http://www.npc.naw.mil/AboutUs/NPC/SurvevPolicy/. 

Survey  Design 

Survey  design  is  both  an  art  and  a  science,  and  this  is  especially  true  for  text 
message  surveys  that  are  limited  to  140  characters  for  both  the  question  and  the 
answer  choices.  When  writing  the  survey,  one  must  be  careful  not  to  use  special 
characters  as  not  all  systems  will  interpret  the  characters  the  same.  Therefore, 
characters  should  be  limited  to  letters,  numbers,  and  the  period  (.). 
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Acronyms  (e.g.,  CO,  POD,  SOP)  are  used  daily  within  Navy  operations,  but 
extreme  caution  must  be  taken  in  using  acronyms  when  creating  survey  questions, 
particularly  for  text  message  surveys.  Some  individuals  may  be  considered 
“heavy-texters”,  who  send  several  messages  daily,  and  they  may  be  familiar  with 
text  message  lingo  (see,  for  example,  http://www.lingo2word.com/index.php), 
but  it  is  possible  that  most  of  those  receiving  the  survey  will  not  be  familiar  with 
this  form  of  short-hand.  To  minimize  confusion  and  potential  miscommunication 
of  the  survey  questions  and  responses,  researchers  should  exercise  caution  in 
using  any  acronym. 

Because  of  the  limited  number  of  characters,  it  may  be  appropriate  to  confine 
answer  choices  to  two  or  three  options.  This  could  possibly  hinder  comparisons 
to  larger  web-  or  paper-based  surveys,  but  it  benefits  both  the  respondent  and  the 
programmer  to  keep  questions  within  the  140  characters  of  a  single  text  message. 
In  addition,  using  different  letters  as  answers  to  each  question  should  be 
considered.  For  example,  answers  to  the  first  question  could  be  A,  B,  C,  and  for 
the  second  question  could  be  X,  Y,  Z.  Realizing  that  message  timing  might 
become  scrambled  (e.g.,  the  respondent  decides  to  expand  on  their  answer  to  a 
previous  question,  thereby  negating  the  easy  flow  of  question-answer-question- 
answer),  researchers  could  still  match  answer  choices  with  the  relevant  question. 

Additionally,  answers  need  to  be  offset  from  the  questions  sufficiently  so  that 
respondents  can  easily  see  what  the  answer  choices  are.  Each  cell  phone 
brand/model  and  provider  handles  and  displays  special  characters  differently,  so 
the  decision  was  made  for  NPRST  Lightning  Polls  to  separate  each  of  the  answers 
from  the  question  with  a  series  of  periods  (“How  satisfied  are  you  with  your  Navy 
job?. . .  a.  Satisfied  . .  .b.  Neither  . . .  c.  Dissatisfied”);  preliminary  tests  showed 
that  multiple  spaces,  hard  returns,  and  special  characters  did  not  format 
consistently  across  phone  models,  whereas  periods  did. 

A  text  message  survey  should  be  short,  precise,  and  quick.  This  can  be 
accomplished  by  restricting  the  number  of  questions  (i.e.,  5  or  less). 

Demographic  data  may  be  gathered  through  other  means  so  that  the  questions 
are  not  part  of  the  text  messaging  survey,  and  then  the  text  message  responses 
can  be  linked  to  that  demographic  data  during  analyses. 

Also  consider  including  the  question  number  as  part  of  the  question  text 
message  (e.g.,  1.,  2.,  3.,...).  A  proxy  progress  bar  may  also  be  included  by 
including  the  question  number  as  well  as  the  total  number  of  questions  (e.g.,  1/3, 
2/3,  3/3)- 

Push  vs  Pull 

There  are  two  techniques  to  be  considered  for  the  survey,  depending  upon  the 
notification  method.  If  cell  phone  numbers  are  not  available  and  a  broadcast 
method  is  used  whereby  the  survey’s  cell  phone  number  is  provided  and  those 
interested  in  participating  are  asked  to  send  a  message  to  that  number  to  start  the 
survey,  this  is  a  “pull”  technique;  the  respondents  are  “pulFing  down  the  survey. 

A  longer  fielding  period  for  “pull”ed  surveys  may  be  needed.  For  example,  if 
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mailing  letters  to  potential  participants,  some  may  be  able  to  start  the  text 
message  survey  a  week  earlier  than  others  (dependent  upon  how  long  the  mail 
takes  to  arrive),  so  the  fielding  period  may  need  to  remain  open  to  account  for 
delays  in  mailing. 

If  a  list  of  cell  phone  numbers  is  already  available  and  all  respondents  will 
begin  at  the  same  time  (perhaps  thereby  having  a  shorter  yet  delayed  fielding 
period),  this  is  a  “push”  technique  where  the  first  question  is  “push”ed  to 
everyone.  If  using  a  panel  (series  of  surveys,  with  known  respondents  and  cell 
phone  numbers),  respondents  may  be  simply  awaiting  the  next  month’s 
questions,  so  the  “push”  technique  is  faster. 

An  issue  to  be  aware  of  when  deciding  between  “push”  vs  “pull”  is  that 
messages  are  handled  in  a  first-in  first-out  manner  by  IMU  Pro  as  well  as  other 
software.  When  pushing  survey  questions  to  a  batch  of  100  respondents,  there 
may  be  a  delay  in  sending  out  second  questions;  the  first  person  in  the  batch  may 
reply  to  the  first  question  and  be  expecting  the  second,  but  must  wait  to  receive  it 
until  the  other  99  people  have  received  the  first  question.  In  small  batches,  this 
delay  may  be  just  a  few  minutes.  In  batches  of  several  thousands  of  respondents, 
the  delay  may  be  more  extreme  (a  day  or  two).  If  using  a  “push”  technique,  it  is 
important  to  consider  doing  a  series  of  smaller  waves  (groups  of  respondents)  so 
that  the  respondent  pool  of  1,000  becomes  10  pools  of  100,  separated  by  a  period 
of  time  such  as  an  hour.  This  will  allow  those  who  are  quick  responders  to  finish 
their  survey  without  waiting  for  delays  in  message  sending. 

It  may  be  possible  to  convert  a  “pull”  survey  (e.g.,  unknown  cell  phone 
numbers  that  normally  would  require  respondents  to  start  as  soon  as  they  are 
notified  by  an  alternate  method)  to  a  “push”  survey  (e.g.,  if  desiring  the  start  of 
the  survey  to  coincide  with  a  particular  event).  An  option  to  consider  would  be  to 
send  a  letter  to  respondents  asking  them  to  access  a  registration  website  prior  to 
the  text  message  survey,  where  respondents  could  register  their  cell  phone 
number  and  demographic  data  at  any  time.  The  text  message  survey  could  then 
be  sent  at  the  same  time  to  all  those  who  have  registered,  regardless  of  the  day 
that  they  received  their  notification  letter  or  registered. 

Notification  process 

All  Navy-wide  surveys  are  required  to  have  a  RCS,  or  license,  as  outlined  in 
OPNAVINST  6300. 8C.  The  instruction  indicates  that  all  Navy  survey 
respondents  be  notified  of  their  rights  prior  to  beginning  a  survey  (Chief  of  Naval 
Operations,  2008).  In  addition,  the  Navy  Survey  Office  who  requests  the  RCS 
may  determine  that  review  of  the  survey  by  an  IRB  is  required  if  the  survey  may 
involve  protection  of  human  subjects  issues,  and  the  IRB  may  require  additional 
consent.  Because  of  the  limited  amount  of  space  in  a  text  message,  respondents 
will  likely  need  to  be  notified  of  this  information  through  alternative  means, 
along  with  notifying  them  of  the  opportunity  to  participate. 
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Part  of  the  requirements  for  Navy  survey  approval  relate  to  confidentiality.  It 
may  be  appropriate  to  reiterate  in  plain  language  within  the  notification  letter 
that  this  confidentiality  includes  their  cell  phone  numbers,  and  that  those 
numbers  will  be  used  only  for  the  survey  and  not  retained  nor  reused. 

Respondents  may  lack  experience  in  completing  text  message  surveys  nor 
realize  that  the  system  is  automated,  so  the  notification  should  also  include  some 
instructions  for  respondents.  For  example,  provide  a  statement  that  respondents 
should  only  provide  the  letter  of  the  answer  instead  of  the  entire  answer,  and  that 
they  may  call  a  separate  phone  number  if  they  have  technical  questions. 

Costs  of  text  messaging  are  dependent  upon  the  cell  phone  plan  that  each 
person  has  chosen.  Consider  notifying  the  respondent  of  the  number  of  text 
messages  that  they  will  use  on  their  monthly  allowance  (both  incoming  and 
outgoing)  when  notifying  them  of  the  opportunity  to  participate. 


Programming 


The  NPRST  Lightning  Polls  were  programmed  with  IMU  Pro  software.  IMU 
Pro  identifies  capabilities  as  Tasks.  There  is  a  Voting  Task,  which  might  be 
effective  for  a  single  question.  This  was  not  tested  since  the  goal  of  the  Lightning 
Poll  was  to  ask  multiple  questions,  preferably  in  a  manner  such  that  a  response  to 
the  first  question  generates  the  second  question,  repeated  until  reaching  the  end 
of  the  survey.  The  Trivia  Task  can  successively  send  questions,  and  therefore  was 
chosen  for  the  NPRST  Lightning  Poll. 

To  begin  the  process,  a  new  trivia  task  was  created  by  clicking  Tasks  on  the 
menu  on  the  left,  and  then  Trivia,  yielding  Figure  2. 


Figure  2.  Adding  a  new  Trivia  Task. 
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Clicking  OK  to  this  screen  presented  an  overview  screen  of  the  task  (Figure  3), 
including  start  and  stop  dates,  cell  phone  number  of  the  SIM  card  being  used, 
current  status,  and  users  who  have  completed  (when  there  are  some). 


Figure  3.  Trivia  Task  overview  screen. 

The  new  task  was  edited  (by  using  the  button  above  the  overview  screen)  to 
allow  creation  of  the  survey.  The  two  key  tabs  of  the  Trivia  Setup  for  a  survey  are 
the  Q&A  Steps  and  Options.  Q&A  Steps  is  where  questions  actually  are  input  into 
the  system.  The  Q&A  Steps  tab  (Figure  4)  was  clicked  and  a  new  step  was  added. 

-i  I  f  f  vr-ry  '^TT  ■  .  — ~~  (9Q£3 


Figure  4.  Adding  questions  to  a  survey. 


10 


IMU  Pro  can  check  for  either  a  correct  text  answer,  a  correct  numeric  answer, 
or  allow  all  answers.  In  a  survey,  respondents  can  select  any  one  of  several 
possible  answers  (e.g.,  “A,”  “B,”  or  “C”)  and  they  may  also  provide  their  response 
in  one  of  many  formats  (e.g.,  “A,”  “a,”  “Agree,”  “agree,”  “AGREE,”  “a.  Agree,”  “A) 
Agree,”  “A. “A.  Go  Navy”);  there  is  no  single  correct  answer  as  there  would  be 
on  a  test,  so  the  easiest  method  of  programming  is  to  select  “Without  correct 
answer”  (Figure  5). 

If"  ■■  nrrr  aaa 


Figure  5.  Adding  a  survey  question  that  has  no  correct  answer. 

The  next  question  to  be  asked— including  the  possible  answers  for  it— was 
input  into  the  “Answer  response”  blank.  Any  additional  questions  can  be  added  in 
the  same  manner. 

Once  all  questions  except  for  the  first  (for  a  “push”ed  survey)  were  input  as 
Q&A  Steps,  the  Options  tab  was  selected  in  order  to  end  the  survey.  A  thank  you 
message  was  added  as  well  (see  Figure  6). 
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Figure  6.  Stopping  the  survey  and  sending  a  thank  you  message. 


Once  everything  is  programmed,  the  subscreen  showed  the  steps  that  were 
programmed,  as  in  Figure  7. 


Add  Trivia  task 


Task  Scheduler  j  VMU  Commands  j 


T  rivia 

T  rivia  name  (shown  in  I  MU  T  ask  list) 
| Trivia  task 


a  Connection  |Modem:  *1901 451 1522 1*1901451 1522)  3 

r  Prefix 


T  rivia  settings 

TOTAL  NUMBER  OF  STEPS:  2  a 

Prizes:  disabled 

Help  command:  "HELP" 

STEP  1 


Step  type:  Without  correct  answer 

Answer  responce:  "2/5.  What  is  the  morale  at  you  command7  a  Low  b.  M» 
Help  response: 

Prize  type:  None 


OK  |  Cancel 


Figure  7.  Trivia  Task  with  steps  programmed. 


To  enable  the  system  to  process  the  answers  and  send  the  questions 
automatically,  the  Trivia  Task  was  activated. 
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Administration 


Once  the  Trivia  Task  was  active,  the  system  was  ready  to  send  out  survey 
questions  automatically.  If  the  survey  is  planned  such  that  respondents  can  start 
the  first  question  at  any  time  (“pull”  technique),  no  further  action  is  needed  to 
start  the  survey  other  than  notifying  those  selected  that  they  may  begin  at  any 
time.  For  “push”ed  surveys,  the  first  question  still  must  be  sent  to  the 
respondents. 

I  mporting  users 

If  the  survey  is  launched  to  all  respondents  at  the  same  time  (“push” 
technique),  respondent  phone  numbers  still  need  to  be  added  to  the  system.  To 
do  this  in  IMU  Pro,  all  cell  phone  numbers  need  to  be  properly  formatted  (with 
no  punctuation  and  a  +1  added  for  CONUS  numbers)  for  importing.  This  can  be 
done  through  Microsoft  Excel  or  similar  software  for  concatenating  +1  to  the 
beginning  of  each  phone  number.  The  properly  formatted  list  of  cell  phone 
numbers  then  needs  to  be  saved  as  a  .txt  file.  For  ease,  and  to  minimize  delays 
that  might  occur  in  reading  from  a  floppy  drive,  save  the  .txt  file  onto  the  hard 
drive  of  the  survey  computer  prior  to  importing  the  phone  numbers. 

To  import  into  IMU  Pro,  select  the  Users  icon  from  the  left  menu  and  then 
select  Users  on  the  menubar,  followed  by  “Import  users.”  The  Import  Users 
subscreen  is  displayed,  as  in  Figure  8;  click  “Open  text  file...”  and  select  the  file 
name,  then  select  Apply  (making  the  numbers  appear  on  the  screen)  and  Close. 


Figure  8. 1  mport  users  subscreen. 
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If  the  survey  is  being  sent  out  in  waves,  consider  adding  a  color  to  the  users  to 
aid  in  tracking  the  waves.  This  must  be  done  individually  for  each  user,  but  is 
worth  the  time  when  different  waves  are  being  sent  out  as  the  list  of  users  can  be 
sorted  by  Color  and  only  those  with  a  particular  color  may  be  easily  selected  (see 
Figure  9).  To  change  user  colors,  right  click  on  the  cell  phone  number  to  display 
the  user  properties,  and  then  use  the  pulldown  arrow  to  select  a  different  color. 


Figure  9.  Users  screen,  with  different  colors  in  use. 

Sending  First  Message  (for  "Push"ed  Surveys) 

With  a  “push”ed  survey,  all  respondents  start  the  survey  by  receiving  the  first 
question  at  roughly  the  same  time  (dependent  upon  how  fast  the  messages  are 
sent  by  the  computer/modem).  To  send  the  first  question  in  IMU  Pro,  select 
those  users  to  receive  it  (if  doing  waves  and  utilizing  colors  to  differentiate 
between  the  waves,  select  all  of  those  with  the  desired  color)  and  right  click  to 
send  message  to  those  users.  IMU  Pro  provides  a  message  indicating  how  many 
users  have  been  selected.  If  this  is  the  correct  number,  select  OK.  Paste  the  text  of 
the  first  message  into  the  box  provided  (see  Figure  10);  if  the  text  is  longer  than 
140  characters,  appropriate  changes  can  be  made  at  this  point.  Be  sure  that  all 
the  answer  choices  are  included,  separated  from  the  question  and  each  other  by 
periods.  Once  the  message  is  correct,  click  send. 
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Figure  10.  Sending  text  messages  to  multiple  phone  numbers. 

Monitoring  Messages  While  the  Survey  is  in  the  Field 

Once  the  survey  has  started,  everything  should  be  automatic  so  only 
occasional  monitoring  of  the  messages  and  computer  is  required.  Clicking  the 
Messages  icon  on  the  left  displays  the  different  folders  for  the  messages  (Figure 
n).  Since  the  Trivia  Task  is  active,  all  incoming  messages  are  automatically 
deposited  in  the  Trivia  folder  (toward  the  bottom  of  the  list  of  folders)  and  not 
the  Inbox.  All  the  outgoing  messages  are  briefly  in  the  Outbox  (queued  to  be  sent) 
before  going  to  the  Sent  folder.  If  numbering  of  the  questions  was  provided, 
sorting  either  the  Outbox  or  the  Sent  folder  by  the  message  itself  can  show  how 
many  will  soon  or  have  already  received  a  particular  question.  If  question  number 
was  not  included,  answers  maybe  sorted  for  each  person  (phone/e-mail). 


Iiilcirti  livu  Muu^viifV.llnliiiiileiM.0  PRO  Ruilil  1069 
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Figure  11. 1  ncoming  messages,  placed  automatically  in  the  Trivia  folder 

for  an  active  Trivia  Task. 
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Occasionally,  messages  have  problems  during  the  sending  process  and  are 
placed  into  the  Unsent  folder.  If  the  intended  cell  phone  numbers  of  these 
messages  are  correct,  you  can  attempt  to  resend  them  by  selecting  the  unsent 
message,  right  clicking,  and  choosing  to  move  it  to  the  Outbox. 

When  the  survey  fielding  period  is  ready  to  close,  go  to  the  Trivia  Task  and 
click  Terminate. 


Analysis 


Exporting  Data 

Once  survey  administration  is  complete  and  the  Trivia  Task  is  terminated, 
messages  must  be  exported  from  IMU  Pro  so  that  they  can  be  analyzed  in  another 
software  program  (e.g.,  Microsoft  Excel,  SPSS,  SAS).  The  key  variables  needed 
are  the  phone  number  (in  order  to  link  between  responses  as  well  as  to  other 
demographic  data,  if  needed),  date /time  (for  ordering  response  messages),  and 
either  the  original  message  text  or  the  Edited/Displayed  message  text  (the 
response).  The  Folder  may  be  useful  as  well  to  differentiate  the  sent  messages 
from  the  incoming,  if  exporting  both  the  questions  and  the  answers  (click  the  All 
Messages  folder  at  the  bottom  of  the  folder  list);  see  Figure  12  for  selections 
within  IMU  Pro. 


Messages  -  Trivia 


I  Phone/Email  1  Pic/Nickname  I  ~  Dale/Tim 


'  D.ate/Time . 


Phone  Number 
l  l  Email  Address 
Original  Message  text 
*1  Edited/Displayed  message  text 
Vi  Folder 
l  Block  type 
Session 
1  Sent  by 

Include  columns  headers  into  output  text  tile 


3  PM  0  c 


Welcome  to  Export  Messages  Wizard 

This  wizard  will  help  you  to  export  messages  into  a  file. 

The  resulting  file  may  be  viewed  directly,  or  it  may  be  imported  into 
another  program  such  as  Microsoft  Excel  or  Access. 

NOTE:  Only  messages  in  the  current  folder  (T rivia)  will  be 
exported.  If  you  want  to  export  messages  from  another  folder,  press 
Cancel  and  then  select  the  relevant  folder.  If  you  want  to  export  all 
messages,  select  the  "All"  folder  before  starting  the  Export  Messages 
Wizard. 

Export  following  columns 


&  How  often 
[3  How  often 
00k 
0A 
0C 
0A 
00 
kd-2 
03 
0D 
0 
0C 
0® 

0d 

0a 

0c 


MODEM-1522 

1522) 


Figure  12.  Exporting  survey  data. 


To  conduct  any  time  analyses  (time  to  complete  first  question,  time  to  survey 
completion,  etc.),  the  outgoing  messages  should  also  be  exported.  This  can  be 
done  from  the  Sent  box.  Alternatively,  All  Messages  (bottom  folder)  can  be 
exported  to  avoid  doing  this  in  two  steps. 
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Messages  are  saved  as  a  tab  delimited  text  file  that  can  be  imported  into 
analysis  software,  as  shown  in  Figure  13. 
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Figure  13.  Exported  text  message  data,  opened  in  Notepad. 


Data  Cleaning 

Almost  all  survey  data  must  be  cleaned  in  some  way  before  being  ready  for 
analysis,  and  text  message  data  is  no  different.  Because  respondents  may  not  be 
constrained  in  their  answers,  there  may  not  be  a  simple  way  of  counting  how 
many  said  “a,”  for  example.  An  easy  way  to  start  cleaning  the  data  is  to  sort  the 
answers  alphabetically.  Those  that  do  not  have  answers  in  the  desired  format  can 
be  modified  before  continuing  with  analyses.  Occasionally,  some  respondents 
may  include  a  letter  of  their  answer  as  well  as  the  word,  for  example  “a. 

Disagree,”  and  the  two  may  be  inconsistent  with  the  responses  possible.  A 
recommendation  would  be  to  select  the  written  answer  (“Disagree”)  instead  of 
the  letter  (“a”),  since  the  respondent  likely  spent  more  time  typing  in  the  written 
answer  than  the  letter. 

In  addition,  some  people  may  have  responses  that  were  too  long  for  a  single 
message,  perhaps  because  the  entire  question  was  included  in  the  response;  these 
may  be  easily  determined  by  looking  for  simultaneous  responses  from  a  single 
user.  The  actual  response  for  these  messages  may  be  at  the  end  of  the  second/last 
message,  at  the  beginning  of  the  first,  or  embedded  in  other  text.  These  data  may 
not  be  easily  modified  after  an  alphabetical  sort  as  discussed  above.  For  cases 
where  there  are  multiple  messages  for  a  single  response,  sorting  by  user  and  time 
of  response  will  make  it  easier  to  determine  the  answer. 
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Analyses  in  SPSS 


There  are  a  number  of  tools  that  can  be  utilized  to  analyze  the  data.  The 
information  below  may  be  adapted  but  is  written  using  the  SPSS  programming 
language  (see  the  SPSS  website  at  www.spss.com  for  more  information). 

Regardless  of  the  analysis  package,  the  text  messaging  data  likely  needs  to  be 
modified  to  make  it  easy  to  analyze.  For  example,  the  date/time  of  the  messages 
are  exported  from  IMU  Pro  as  “5/14/2008  1:10:09  PM”,  which  may  not  be  the 
needed  format  for  computing  time  differentials.  Figure  14  provides  some  example 
syntax  for  converting  the  date/time  of  a  message  into  the  date /time  formats  used 
within  SPSS. 


***  Computing  new  variables  for  month,  day,  and  year. 

***  dtout  is  the  variable  name  given  to  the  message  dates/times  when 
importing  the  data. 

compute  outlmo=number(substr(dtout,l,(index(dtout,7'))-l),F2.0). 
compute  outlda=number(substr(dtout, indexf dtout, y)  +  l, rindexfdtout, 
index(dtout,7')-l),f2.0). 
compute  outlyr=2008. 


***  Computing  new  variables  for  hour  (24-hour  format),  minute,  and  second. 

compute  outlhr=number(substr(dtout,index(dtout,'  ')  +  l,index(dtout,':')- 
index(dtout,'  ')-l),f2.0). 

if  substr(dtout,index(dtout,'M')-l,l)='Pl  outlhr=outlhr+12. 
if  outlhr=12  outlhr=0. 
if  outlhr=24  outlhr=12. 

compute  outlmn=number(substr(dtout,index(dtout,':l)  +  l,rindex(dtout,':1)- 
index(dtout, f2.0). 

compute  outlss=number(substr(dtout, rindex(dtout, 2), f2.0). 


***  Combining  new  variables  into  SPSS  date  and  time  formats, 
compute  msgdate=date.mdy(outlmo,outlda,outlyr). 


Figure  14.  Sample  SPSS  syntax  to  convert  message  date/  time. 

Once  the  dates  and  times  are  in  SPSS  format,  completion  times  can  be  found 
using  the  datediff  function. 

It  may  be  useful  to  number  each  message  (line  of  data).  For  example,  if  the 
data  has  been  cleaned  so  that  there  is  one  line  of  data  for  each  survey  answer, 
then  analyses  for  question  3  will  always  be  on  the  third  line  of  data  for  each 
person.  Sorting  by  message  date/time  is  a  proxy  for  this  (the  first  answer  is  the 
message  with  the  oldest  date /time),  but  having  a  count  of  how  many  responses 
per  person  might  be  useful.  Figure  15  provides  SPSS  syntax  for  easily  numbering 
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each  line  of  data  per  person.  If  response  times  are  a  concern,  questions  and 
answers  will  need  to  be  merged  together,  so  answers  should  be  numbered  with 
even  numbers  while  questions  should  be  numbered  with  odd  numbers. 


***  Computing  message  numbers  (for  ease  in  analyses  and  when  combining 
questions  data  with  answers  data). 

***  phonenum  is  the  phone  number,  msgdate  and  msghour  were  created  prior 
and  are  the  message's  date  and  time. 

***  If  making  answers  the  even  numbered  messages  and  questions  the  odd 
numbered  messages,  change  the  lag  function  to  be  +2  and  have  the 

answers  start  with  a  msgnum=2. 

sort  cases  by  phonenum(a)  msgdate(a)  msghour(a). 

compute  msgnum=l. 

execute. 

if  serial^ lag( serial,  1)  msgnum=lag(msgnum,l)  +  l. 

execute. 


Figure  15.  Sample  SPSS  syntax  to  create  a  message  number. 

It  may  be  most  useful  to  have  one  line  of  data  per  person  instead  of  several 
lines.  This  can  easily  be  done  in  the  newer  versions  of  SPSS  by  using  the 
CASESTOVARS  command;  sort  by  phone  number  and  message  number 
beforehand  to  ensure  that  messages  are  in  a  consistent  order. 

Once  there  is  a  single  line  of  data  for  each  person,  demographic  variables  that 
may  exist  in  a  separate  data  base  (for  example,  the  database  with  everyone  who 
registered  to  participate)  can  be  merged  using  the  phone  number  as  the  common 
element  between  the  databases. 


Future  Directions 


Research  on  Navy  text  message  surveys  has  just  begun.  The  lessons  learned 
presented  here  are  based  upon  the  two  Lightning  Polls  conducted  thus  far.  Both 
were  conducted  as  “push”  surveys,  with  the  second  one  also  being  separated  into 
smaller  waves  across  three  months.  Additional  lessons  may  be  learned  through  a 
“pull”ed  text  message  survey,  and  opportunities  for  conducting  a  “pulT’ed  survey 
are  currently  being  considered. 
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Research  remains  to  be  done  on  the  comparability  of  results  between  a  text 
message  survey  and  a  more  traditional  method  such  as  web-based.  The  current 
Lightning  Poll  mirrors  questions  asked  on  other  web-based  surveys,  but 
responses  have  not  yet  been  analyzed.  An  issue  that  may  be  fundamental  regards 
the  changes  required  to  convert  a  survey  question  and  answers  into  the  140 
characters  required  for  a  text  message. 

There  may  also  be  unique  questions  or  topics  that  can  be  satisfied  best  with  a 
text  message  survey  (e.g.,  usefulness  of  text  messaging  as  a  communication  tool) 
while  other  topics  (e.g.,  sexual  harassment)  may  not  be  suitable  or  too  sensitive 
for  text  messaging;  future  research  should  investigate  these  issues  as  well.  This 
issue  may  also  be  demographically  based,  with  some  populations  more  likely  to 
respond  via  text  message  on  specific  topics  than  through  traditional  survey 
methods.  Future  research  should  investigate  what  topics  and/or  populations  are 
most  suitable  for  text  message  surveys. 

Another  possible  future  direction  for  text  message  surveys  would  be  to  create 
a  panel.  This  would  allow  for  a  single  notification  of  respondents,  but  survey 
questions  would  be  asked  periodically.  This  however  does  present  the  challenge 
of  refreshing  the  panel  periodically. 

The  surveys  as  programmed  here  use  the  Trivia  Task,  but  IMU  Pro  also 
includes  a  Voting  Task,  which  allows  for  one  question  and  calculates  responses 
automatically.  While  this  would  likely  not  be  a  scientific  survey  with  weighted 
results,  it  could  be  useful  and  should  be  tested  at  some  point  in  the  future. 
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